Fairness assessment for deep generative models

ABSTRACT

A computer-implemented method, a computer program product, and a computer system for assessing fairness of a deep generative model. A computer system receives a user defined fairness criterion for the deep generative model. A computer system probes the deep generative model to produce samples for a target output. A computer system evaluates the samples for the fairness of the deep generative model, according to the user defined fairness criterion. A computer system produces a set of recommendations for modifying the deep generative model to meet the user defined fairness criterion, in response to determining that the deep generative model does not meet the user defined fairness criterion. In response to determining that the deep generative model is to be modified, a computer system applies at least one subset of the recommendations to the deep generative model. A computer system updates the deep generative model.

BACKGROUND

The present invention relates generally to artificial intelligence (AI)and machine learning, and more particularly to deep generative models,AI fairness, automated artificial intelligence (AutoAI), and machinelearning applications.

An application for medical diagnosis may need data such as medicalrecords for patients or magnetic resonance imaging (MRI) images. Due tothe privacy concerns and sensitivity of such data, an applicationdeveloper will likely utilize a deep generative model to generate suchsynthetic data. However, deep generative models are often trained onlimited data and require computational resource and expertise. On theother hand, deep generative models sourced from third parties may notproduce data that obeys fairness criteria. For example, medical recordsmay not be fairly produced across different groups.

SUMMARY

In one aspect, a computer-implemented method for assessing fairness of adeep generative model is provided. The computer-implemented methodincludes receiving a user defined fairness criterion for the deepgenerative model. The computer-implemented method further includesprobing the deep generative model to produce samples for a targetoutput. The computer-implemented method further includes evaluating thesamples for fairness of the deep generative model, according to the userdefined fairness criterion. The computer-implemented method furtherincludes, in response to determining that the deep generative model doesnot meet the user defined fairness criterion, producing a set ofrecommendations for modifying the deep generative model to meet the userdefined fairness criterion. The computer-implemented method furtherincludes, in response to determining that the deep generative model isto be modified, applying at least one subset of the recommendations tothe deep generative model. The computer-implemented method furtherincludes updating the deep generative model.

In another aspect, a computer program product for assessing fairness ofa deep generative model is provided. The computer program productcomprises a computer readable storage medium having program instructionsembodied therewith, and the program instructions are executable by oneor more processors. The program instructions are executable to: receivea user defined fairness criterion for the deep generative model; probethe deep generative model to produce samples for a target output;evaluate the samples for fairness of the deep generative model,according to the user defined fairness criterion; in response todetermining that the deep generative model does not meet the userdefined fairness criterion, produce a set of recommendations formodifying the deep generative model to meet the user defined fairnesscriterion; in response to determining that the deep generative model isto be modified, apply at least one subset of the recommendations to thedeep generative model; and update the deep generative model.

In yet another aspect, a computer system for assessing fairness of adeep generative model is provided. The computer system comprises one ormore processors, one or more computer readable tangible storage devices,and program instructions stored on at least one of the one or morecomputer readable tangible storage devices for execution by at least oneof the one or more processors. The program instructions are executableto receive a user defined fairness criterion for the deep generativemodel. The program instructions are further executable to probe the deepgenerative model to produce samples for a target output. The programinstructions are further executable to evaluate the samples for fairnessof the deep generative model, according to the user defined fairnesscriterion. The program instructions are further executable to, inresponse to determining that the deep generative model does not meet theuser defined fairness criterion, produce a set of recommendations formodifying the deep generative model to meet the user defined fairnesscriterion. The program instructions are further executable to, inresponse to determining that the deep generative model is to bemodified, apply at least one subset of the recommendations to the deepgenerative model. The program instructions are further executable toupdate the deep generative model.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a systematic diagram illustrating a system for assessingfairness of a deep generative model, in accordance with one embodimentof the present invention.

FIG. 2 is a flowchart showing operational steps for assessing fairnessof a deep generative model, in accordance with one embodiment of thepresent invention.

FIG. 3 is a diagram illustrating components of a computing device orserver, in accordance with one embodiment of the present invention.

FIG. 4 depicts a cloud computing environment, in accordance with oneembodiment of the present invention.

FIG. 5 depicts abstraction model layers in a cloud computingenvironment, in accordance with one embodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention disclose a system that anapplication developer employs to assess the fairness of a pretraineddeep generative model prior to blind adoption of a pretrained model.

Fairness is a property of a pretrained deep generative model to treatall entities equally, and it may be considered for either an individualor a group. When considering for an individual, similar individuals mustbe treated similarly. When considering for a group, a user-specifieddefinition of a group is used and it is ensured that model outputs aresimilar across groups (typically using some statistical measures).Embodiments of the present invention consider group fairness.

FIG. 1 is a systematic diagram illustrating system 100 for assessingfairness of a deep generative model, in accordance with one embodimentof the present invention. System 100 is implemented on one or morecomputing devices or servers. A computing device or server is describedin more detail in later paragraphs with reference to FIG. 3 . System 100may be implemented in a cloud computing environment. The cloud computingenvironment is described in more detail in later paragraphs withreference to FIG. 4 and FIG. 5 .

System 100 for assessing fairness of a deep generative model includesfairness assessor 110. From input 130, fairness assessor 110 receivesuser defined fairness criterion 140 and pretrained deep generative model150. Pretrained deep generative model 150 can generate syntheticsamples. For example, pretrained deep generative model 150 can producemagnetic resonance imaging (MRI) images, and pretrained deep generativemodel 150 can produce human portraits. To define user defined fairnesscriterion 140, a user specifies a fairness criterion for the syntheticsamples. User defined fairness criterion 140 can be defined with respectto some protected attributes or can be specified via samples.

Pretrained deep generative model 150 can be probed to produce syntheticsamples for the target output, and the obtained samples can be evaluatedfor fairness by using a classifier for example. In a scenario wherepretrained deep generative model 150 generates synthetic samplesX~P_(data) (where P_(data) is the data distribution), pretrained deepgenerative model 150 is trained such that, for a given latentrepresentation z~P_(sample) (where P_(sample) is the pre-specifiedsample distribution), G (z) obeys P_(data) (where G (z) is amathematical notation of a deep generative model). The latentrepresentation includes a noise component and can optionally includestructured inputs such as labels.

A user of pretrained deep generative model 150 may specify a fairnesscriterion for generation of synthetic samples with respect to someprotected attributes; for example, the fairness criterion requires thatgeneration of patient records is fair across all groups of patients.

In an embodiment, a fairness criterion is a classifier-defined usercriterion. A user provides an application programming interface (API)access to a classifier that can predict the distributionalcharacteristics for the protected feature. For the purpose ofsimplicity, a following scenario is considered: a protected attribute tis binary (t ∈ {0,1}), and a classifier ƒ: X → t can predict theattribute for an arbitrary sample. Assuming that a user prefers abalanced distribution across elements of t, a simple fairness criterioncan be defined as:

E_(Z ∼ P_(sample))[P(f(G(z)) = 0)] = E_(Z ∼ P_(sample))[P(f(G(z)) = 1)]

The fairness criterion as expressed above states that, under randominput samples, the deep generative model (G) is expected to produceoutputs which satisfy the property represented by 0 and the propertyrepresented by 1 with equal probability.

In another embodiment, a fairness criterion is a sample-defined usercriterion. The sample-defined user criterion may involves usingpre-labelled samples to train an ad-hoc classifier that can subsequentlybe used to assess fairness; for example, an ad-hoc classifier whichpredicts “skin tone” is trained and used for assessing fairness ofsamples. Alternatively, the sample-defined user criterion may involvesby merely comparing (empirically) the distributional statistics of thesample distribution and the desired target distribution; for example, auser may provide a small set of samples of “smiling faces” which areuniformly distributed across a criterion like skin tone. In such a case,system 100 for assessing fairness of a deep generative model can comparethe sample mean, median, or other statistics; alternatively, system 100can use unsupervised techniques to compare clusters across the twodistributions.

Fairness assessor 110 assesses the fairness of pretrained deepgenerative model 150, as per user defined fairness criterion 140. Forinstance, fairness assessor 110 samples from pretrained deep generativemodel 150; fairness assessor 110 uses a classifier to evaluate thefairness or explore the latent space to evaluate fairness.

In one embodiment, fairness assessor 110 assesses the samples with aclassifier. In a scenario where an externally sourced classifier isavailable to help assign protected attributes, pretrained deepgenerative model 150 may be probed for producing few thousands ofsamples and the classifier can be used to statistically assess fairnessproperties of the produced samples. Any classical machine learningclassifier can be embodied in fairness assessor 110.

In another embodiment, fairness assessor 110 traverses the latent space.In order to assess the fairness with respect to the latent space,fairness assessor 110 can explore the neighborhood of different samples.Geometric properties can be used to evaluate whether there are biases ina particular direction or whether the fairness is preserved in certainsubspaces or regions within the latent space. Often, pretrained deepgenerative model 150 incorporates a structured input where a part of theinput is fed via a random seed while the other part is defined byattributes (e.g., “smile” or “glasses” in human portraits). In suchcases, other techniques can be used for fairness assessment. Forexample, fairness assessor 110 can assess group fairness with respect toeach of these attributes. Given structured inputs, fairness assessor 110can also compare the output of pretrained deep generative model 150 fortwo different conditional inputs for a variety of random seeds.

System 100 for assessing fairness of a deep generative model furtherincludes deep generative model modifier 120. Given information generatedby fairness assessor 110, deep generative model modifier 120 analyzeschanges needed to be made to pretrained deep generative model 150 andensures pretrained deep generative model 150 to meet user definedfairness criterion 140. Deep generative model modifier 120 may applysome of these changes directly to pretrained deep generative model 150.

System 100 for assessing fairness of a deep generative model has output160. Output 160 includes fairness report 170. Fairness report 170 isgenerated by fairness assessor 110. In fairness report 170, fairnessassessor 110 summarizes the fairness assessment done by fairnessassessor 110. Output 160 further includes recommendations 180 formodifying the deep generative model. Recommendations 180 is generated bydeep generative model modifier 120. In recommendations 180, deepgenerative model modifier 120 provides a user with recommendations formodifying pretrained deep generative model 150 to meet user definedfairness criterion 140. Recommendations 180 may be used by the user tomanually modify pretrained deep generative model 150. Output 160 furtherincludes updated deep generative model 190. Updated deep generativemodel 190 is generated by deep generative model modifier 120. Deepgenerative model modifier 120 applies all or a subset of recommendations180 to modify pretrained deep generative model 150. Through themodifications, deep generative model modifier 120 generates updated deepgenerative model 190 and improves the fairness of the generation processof the deep generative model.

In one embodiment, deep generative model modifier 120 recommends toreweight inputs of pretrained deep generative model 150. The inputs topretrained deep generative model 150 provide one means to modify theoutput distributions. One simple strategy to ensure fairness is toreconfigure the input distribution as per user defined fairnesscriterion 140. For example, if the initial distribution is standardnormal which led to imbalance classes within the output, deep generativemodel modifier 120 may prescribe a new reweighted distribution such as amixture of Gaussians to satisfy the user criterion.

In another embodiment, deep generative model modifier 120 retrains theparameters of pretrained deep generative model 150. Deep generativemodel modifier 120 can retrain parts of the architecture of deepgenerative model modifier 120 or introduce new architecture componentsto ensure the fairness criterion to be satisfied. In the traditionallytrained deep generative model modifier 120, initial layers help indisentangling the latent space while the later ones help improve thequality of generated samples. Therefore, deep generative model modifier120 may retrain the selective layers, such as the ones responsible fordisentangling features to reconfigure the latent space. When suchflexibility is not readily available, deep generative model modifier 120may include additional layers as a preprocessing module to achieve thesame goal.

FIG. 2 is a flowchart showing operational steps for assessing fairnessof a deep generative model, in accordance with one embodiment of thepresent invention. The operational steps are implemented by a system(e.g., system 100 shown in FIG. 1 ) for assessing fairness of a deepgenerative model. The system for assessing fairness of a deep generativemodel is hosted by one or more computing devices or servers. A computingdevice or server is described in more detail in later paragraphs withreference to FIG. 3 . The operational steps may be implemented in acloud computing environment. The cloud computing environment isdescribed in more detail in later paragraphs with reference to FIG. 4and FIG. 5 .

At step 201, the one or more computing devices or servers receive a deepgenerative model which generates samples. In the embodiment shown inFIG. 1 , fairness assessor 110 in system 100 (hosted by the one or morecomputing devices or servers) receives pretrained deep generative model150.

At step 202, the one or more computing devices or servers receive a userdefined fairness criterion for the deep generative model. The userdefined fairness criterion may specify some target output, for example,generation of samples of MRI images with tumor or generation of sampleswhich are fair with respect to a certain group of patients. In theembodiment shown in FIG. 1 , fairness assessor 110 in system 100 (hostedby the one or more computing devices or servers) receives user definedfairness criterion 140.

At step 203, the one or more computing devices or servers probe the deepgenerative model to produce the samples for a target output. In theembodiment shown in FIG. 1 , fairness assessor 110 probes pretraineddeep generative model 150 to produce the samples.

At step 204, the one or more computing devices or servers evaluate thesamples for fairness of the deep generative model, according to the userdefined fairness criterion. In the embodiment shown in FIG. 1 , fairnessassessor 110 evaluates the samples produced at step 203.

At step 205, the one or more computing devices or servers output areport summarizing fairness analysis for the deep generative model. Inthe embodiment shown in FIG. 1 , fairness assessor 110 outputs fairnessreport 170 which summarizes the fairness assessment done by fairnessassessor 110.

At step 206, the one or more computing devices or servers determinewhether the deep generative model meets the user defined fairnesscriterion. In the embodiment shown in FIG. 1 , deep generative modelmodifier 120 implements this step. In response to determining that thedeep generative model meets the user defined fairness criterion (YESbranch of decision block 206), the one or more computing devices orservers end the operational steps.

In response to determining that the deep generative model does not meetthe user defined fairness criterion (NO branch of decision block 206),at step 207, the one or more computing devices or servers produce a setof modification recommendations for the deep generative model, based onthe report. Given the report obtained at step 205, the one or morecomputing devices or servers analyzes the deep generative model andproduces a set of modification recommendations which help deploy thedeep generative model as per the user fairness criterion. The one ormore computing devices or servers recommend to reweight inputs of thedeep generative model. Alternatively, the one or more computing devicesor servers recommend to retrain parts of the deep generative model orinclude additional parameters for disentangling latent vectors as peruser criterion. In the embodiment shown in FIG. 1 , deep generativemodel modifier 120 receives information generated by fairness assessor110 and recommends changes needed to be made to pretrained deepgenerative model 150 to meet user defined fairness criterion 140.

At step 208, the one or more computing devices or servers determinewhether to modify the deep generative model. The one or more computingdevices or servers may determine whether owners of the deep generativemodel permit the modification of the deep generative model. In theembodiment shown in FIG. 1 , deep generative model modifier 120determines whether to modify the deep generative model.

In response to determining that the deep generative model is to bemodified (YES branch of decision block 208), at step 209, the one ormore computing devices or servers apply at least one subset of themodification recommendations to the deep generative model, to updatedeep generative model. In the embodiment shown in FIG. 1 , deepgenerative model modifier 120 applies all or a subset of recommendations180 to pretrained deep generative model 150 and generates updated deepgenerative model 190. In response to determining that the deepgenerative model is not to be modified (NO branch of decision block208), the one or more computing devices or servers end the operationalsteps.

The one or more computing devices or servers iterate steps 203-209. Theone or more computing devices or servers iterate evaluation of thesamples and modification of the deep generative model until the deepgenerative model meets the user defined fairness criterion.

FIG. 3 is a diagram illustrating components of computing device orserver 300, in accordance with one embodiment of the present invention.It should be appreciated that FIG. 3 provides only an illustration ofone implementation and does not imply any limitations; differentembodiments may be implemented.

Referring to FIG. 3 , computing device or server 300 includesprocessor(s) 320, memory 310, and tangible storage device(s) 330. InFIG. 3 , communications among the above-mentioned components ofcomputing device or server 300 are denoted by numeral 390. Memory 310includes ROM(s) (Read Only Memory) 311, RAM(s) (Random Access Memory)313, and cache(s) 315. One or more operating systems 331 and one or morecomputer programs 333 reside on one or more computer readable tangiblestorage device(s) 330.

Computing device or server 300 further includes I/O interface(s) 350.I/O interface(s) 350 allows for input and output of data with externaldevice(s) 360 that may be connected to computing device or server 300.Computing device or server 300 further includes network interface(s) 340for communications between computing device or server 300 and a computernetwork.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A nonexhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the C programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user’s computer, partly on the user’s computer, as astand-alone software package, partly on the user’s computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user’scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, fieldprogrammable gate arrays(FPGA), or programmable logic arrays (PLA) may execute the computerreadable program instructions by utilizing state information of thecomputer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a computer, or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be accomplished as one step, executed concurrently,substantially concurrently, in a partially or wholly temporallyoverlapping manner, or the blocks may sometimes be executed in thereverse order, depending upon the functionality involved. It will alsobe noted that each block of the block diagrams and/or flowchartillustration, and combinations of blocks in the block diagrams and/orflowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts orcarry out combinations of special purpose hardware and computerinstructions.

It is to be understood that although this disclosure includes a detaileddescription on cloud computing, implementation of the teachings recitedherein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice’s provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider’s computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider’s applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes.

Referring now to FIG. 4 , illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 includes one or morecloud computing nodes 10 with which local computing devices are used bycloud consumers, such as mobile device 54A, desktop computer 54B, laptopcomputer 54C, and/or automobile computer system 54N may communicate.Nodes 10 may communicate with one another. They may be grouped (notshown) physically or virtually, in one or more networks, such asPrivate, Community, Public, or Hybrid clouds as described hereinabove,or a combination thereof. This allows cloud computing environment 50 tooffer infrastructure, platforms and/or software as services for which acloud consumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N areintended to be illustrative only and that computing nodes 10 and cloudcomputing environment 50 can communicate with any type of computerizeddevice over any type of network and/or network addressable connection(e.g., using a web browser).

Referring now to FIG. 5 , a set of functional abstraction layersprovided by cloud computing environment 50 (FIG. 4 ) is shown. It shouldbe understood in advance that the components, layers, and functionsshown in FIG. 5 are intended to be illustrative only and embodiments ofthe invention are not limited thereto. As depicted, the following layersand corresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provide prearrangement for, and procurement of, cloudcomputing resources for which a future requirement is anticipated inaccordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and function 96. Function 96 in the presentinvention is the functionality of fairness assessment for deepgenerative models.

What is claimed is:
 1. A computer-implemented method for assessingfairness of a deep generative model, the method comprising: receiving auser defined fairness criterion for the deep generative model; probingthe deep generative model to produce samples for a target output;evaluating the samples for the fairness of the deep generative model,according to the user defined fairness criterion; in response todetermining that the deep generative model does not meet the userdefined fairness criterion, producing a set of recommendations formodifying the deep generative model to meet the user defined fairnesscriterion; in response to determining that the deep generative model isto be modified, applying at least one subset of the recommendations tothe deep generative model; and updating the deep generative model. 2.The computer-implemented method of claim 1, further comprising:iterating evaluation of the samples and modification of the deepgenerative model, until the deep generative model meets the user definedfairness criterion.
 3. The computer-implemented method of claim 1,further comprising: outputting a report summarizing fairness analysisfor the deep generative model.
 4. The computer-implemented method ofclaim 1, wherein the recommendation is used by an user to modify thedeep generative model.
 5. The computer-implemented method of claim 1,further comprising: determining whether an owner of the deep generativemodel permits modification of the deep generative model; in response todetermining that the owner of the deep generative model permits themodification, applying the at least one subset of the recommendations tothe deep generative model.
 6. The computer-implemented method of claim1, wherein the deep generative model is pretrained.
 7. Thecomputer-implemented method of claim 1, wherein, for evaluating thesamples according to the user defined fairness criterion, a userprovides an application programming interface access to a classifierthat predicts distributional characteristics for a protected feature. 8.A computer program product for assessing fairness of a deep generativemodel, the computer program product comprising a computer readablestorage medium having program instructions embodied therewith, theprogram instructions executable by one or more processors, the programinstructions executable to: receive a user defined fairness criterionfor the deep generative model; probe the deep generative model toproduce samples for a target output; evaluate the samples for thefairness of the deep generative model, according to the user definedfairness criterion; in response to determining that the deep generativemodel does not meet the user defined fairness criterion, produce a setof recommendations for modifying the deep generative model to meet theuser defined fairness criterion; in response to determining that thedeep generative model is to be modified, apply at least one subset ofthe recommendations to the deep generative model; and update the deepgenerative model.
 9. The computer program product of claim 8, furthercomprising the program instructions executable to: iterate evaluation ofthe samples for the fairness and modification of the deep generativemodel, until the deep generative model meets the user defined fairnesscriterion.
 10. The computer program product of claim 8, furthercomprising the program instructions executable to: output a reportsummarizing fairness analysis for the deep generative model.
 11. Thecomputer program product of claim 8, wherein the recommendation is usedby an user to modify the deep generative model.
 12. The computer programproduct of claim 8, further comprising: determine whether an owner ofthe deep generative model permits modification of the deep generativemodel; in response to determining that the owner of the deep generativemodel permits the modification, apply the at least one subset of therecommendations to the deep generative model.
 13. The computer programproduct of claim 8, wherein the deep generative model is pretrained. 14.The computer program product of claim 8, wherein, for evaluating thesamples according to the user defined fairness criterion, a userprovides an application programming interface access to a classifierthat predicts distributional characteristics for a protected feature.15. A computer system for assessing fairness of a deep generative model,the computer system comprising one or more processors, one or morecomputer readable tangible storage devices, and program instructionsstored on at least one of the one or more computer readable tangiblestorage devices for execution by at least one of the one or moreprocessors, the program instructions executable to: receive a userdefined fairness criterion for the deep generative model; probe the deepgenerative model to produce samples for a target output; evaluate thesamples for the fairness of the deep generative model, according to theuser defined fairness criterion; in response to determining that thedeep generative model does not meet the user defined fairness criterion,produce a set of recommendations for modifying the deep generative modelto meet the user defined fairness criterion; in response to determiningthat the deep generative model is to be modified, apply at least onesubset of the recommendations to the deep generative model; and updatethe deep generative model.
 16. The computer system of claim 15, furthercomprising the program instructions executable to: iterate evaluation ofthe samples for the fairness and modification of the deep generativemodel, until the deep generative model meets the user defined fairnesscriterion.
 17. The computer system of claim 15, further comprising theprogram instructions executable to: output a report summarizing fairnessanalysis for the deep generative model.
 18. The computer system of claim15, wherein the recommendation is used by an user to modify the deepgenerative model.
 19. The computer system of claim 15, furthercomprising: determine whether an owner of the deep generative modelpermits modification of the deep generative model; in response todetermining that the owner of the deep generative model permits themodification, apply the at least one subset of the recommendations tothe deep generative model.
 20. The computer system of claim 15, whereinthe deep generative model is pretrained.